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Abstract 

An experimental field study at a German university was conducted in order to test the effectiveness of an 
integrated learning environment to improve the acquisition of knowledge about empirical research methods. The 
integrated learning environment was based on the combination of instruction-oriented and problem-oriented 
design principles and consisted of twelve worked examples. An elaboration intervention was administered as 
instructional support. 

The effectiveness of the learning environment both with and without the elaboration intervention was assessed 
using knowledge application tasks (near and far transfer), which were applied after the training phase. In 
addition, student’s self-reports on mindfulness (Salomon & Globerson, 1987) were collected. The training was 
implemented into the regular curriculum. The participants were advanced students in educational science. Both 
experimental groups (with elaboration intervention: n = 26; without elaboration intervention n = 27) clearly 
outperformed the control group (n = 17) in the knowledge application tasks. In order to (successfully) foster 
transferable applicable knowledge, instructional support provided via the elaboration intervention was in fact 
necessary. Furthermore, the self-reports of students in the experimental group with elaboration intervention 
showed higher mindfulness scores than those without it. Our results indicate that the integrated learning 
environment developed in this study can be implemented to improve the acquisition of knowledge about 
empirical research methods both effectively and efficiently. 

Keywords: worked examples, integrated learning environment, instructional support, knowledge application, 
knowledge transfer, mindfulness 

1. Introduction 

Many students of the social sciences experience difficulties in understanding concepts of statistics. Frequently, a 
deeper understanding of the principles of statistical methods is lacking. As a result, students tend to apply 
erroneous concepts to the interpretation of statistical results (e.g., Castro Sotos, Vanhoof, van den Noortgate, & 
Onghena 2007; 2009). For instance, several studies have shown students’ deficits in their ability to interpret 
statistical data and empirical results in correlation analyses (Krause, 2007; Krause & Stark 2010). Here, learning 
without reflecting the content of what is being learned, often accompanied by motivational deficits, has been 
identified as an antecedent of knowledge gaps (e.g., Stark & Mandl, 2000; Stark, Puhl, & Krause, 2009; Tyroller, 
2005). 

Furthermore, students often acquire inert knowledge (Gruber & Renkl, 2000). Inert knowledge at best is 
accessible in exam situations, but cannot be applied to solve complex problems in practice. Problems of this type 
are frequently observed in instruction-oriented learning environments (Grasel & Mandl, 1999; Reinmann & 
Mandl, 2006), a teaching method that is common among the traditional teaching of research methods in the 
social sciences. Students therefore often lack essential competencies in applying their knowledge of quantitative 
statistics. 

The goal of this study was to investigate to what extent the outlined problem can be counteracted by using an 
innovative integrated learning environment (Reinmann & Mandl, 2006). For this purpose, a learning 
environment based on the integration of both instruction-oriented and problem-oriented design principles was 
developed. This learning environment depicts several real findings of an experimental study by Tyroller (2005). 
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Learning tasks were developed as authentic and relevant worked examples. The worked examples consist of 
problems regarding the use of statistical methods as well as the interpretation of results and the respective 
detailed solutions. By means of studying this learning environment, the students’ acquisition of applicable 
knowledge to interpret empirical results was to be developed further. 

In order to support the effectiveness of the integrated learning environment, we developed an elaboration 
intervention to facilitate working with the provided material. The intervention included an advance organizer 
(Ausubel, 1960) that structures working in advance and a short phase of cognitive modelling (Cognition and 
Technology Group at Vanderbilt [CGTV], 1992; 1993) which explains how to use the advance organizer. 

We conducted an experimental field study within the setting of advanced seminars in statistics for the 
educational sciences; our aim was to systematically examine both the effectiveness of the learning environment 
and the elaboration intervention. By testing the latter, we intended to contribute to the research regarding the 
effectiveness of instructional support during learning with complex worked examples (e.g., Atkinson, Derry, 
Renkl, & Wortham, 2000; Hilbert & Renkl, 2008; Renkl & Atkinson, 2007; Stark, 1999; 2000; 2001). 

1.1 Effectiveness of Learning with Worked Examples 

Learning with worked examples is a method of instruction-oriented learning approaches (e.g., Reinmann & 
Mandl, 2006). It has mainly been applied in well-structured domains (a comprehensive review on the subject is 
provided by Atkinson et al., 2000; q.v. Hilbert & Renkl, 2008; Reimann, 1997; Renkl, 2001a; Renkl & Atkinson, 
2002; Stark, 1999; 2000; 2001 for further results). 

When learning with worked examples, the learner is presented with the problem, the problem-solving steps as 
well as the solution itself. The task for the learner is to elaborate these example solutions (Stark, 1999; 2001; 
Stark, Kopp, & Fischer, 2011) and to comprehend the solution. Working on several isomorph worked examples 
is supposed to support the acquisition of competencies for problem solving, thus primarily improving applicable 
knowledge (for taxonomies of knowledge see De Jong & Ferguson-Hessler, 1996). The knowledge base thus 
acquired can facilitate future transfer tasks (Stark, 1999). 

The effectiveness of worked examples in well-structured domains is well documented (e.g., Atkinson et ah, 2000; 
Stark, 1999; 2000; 2001). For instance, learning with worked examples supports the acquisition of applicable 
knowledge in mathematics (e.g., Durkin & Rittle-Johnsson, 2012; Hilbert, Renkl, Kessler, & Reiss, 2008; Renkl, 
2001b; 2005; Renkl, Schworm, & vom Hofe, 2001). Furthermore, our research group found empirical evidence 
for the effectiveness of worked examples in the acquisition of conceptual and applicable knowledge regarding 
the domain of empirical research methods (e.g., improving the mathematical understanding of correlation 
analysis, Krause & Stark, 2010). 

There are several mechanisms involved in the effectiveness of worked examples. Since learners are not required 
to find the solution by themselves, they are able to focus their attention towards understanding the solution and 
the problem-solving steps (Renkl, Gruber, Weber, Lerche, & Schweizer, 2003). Due to lower cognitive 
processing efforts, more cognitive resources are available for the acquisition of problem solving schemata (Paas, 
Renkl, & Sweller, 2004; Sweller, 2005). This in turn facilitates transfer later on. Within contexts of initial 
knowledge acquisition, worked examples are thus thought to be superior to conventional problem-solving tasks 
(q.v. cognitive load theory; Sweller, 1988). The findings of Paas and van Merrienboer (1994) as well as Ward 
and Sweller (1990) provide empirical support for this notion. 

The ability of learners to self-explain the principles behind the example solutions is another factor that 
considerably influences the effectiveness of learning with worked examples ( self-explanations; Chi, Bassok, 
Lewis, Reimann, & Glaser, 1989; Chi, De Leeuw, Chiu, & La Vancher, 1994; Renkl, 1997). Self-explanations 
and example elaborations (Stark, 1999) which support learning effectively are characterized by putting an 
emphasis on structure, meaning that learners explicitly focus on the common concepts and principles underlying 
the problem-solving steps within different worked examples. This enables the learners to anticipate the structure 
of upcoming tasks. Less effective example elaborations are paraphrasing or simply re-reading the worked 
examples (Stark, 1999). 

However, despite empirical evidence for learning effectiveness ( worked example effect, Clark & Mayer, 2003) 
worked examples cannot be assumed to be generally superior to other types of learning tasks. Worked examples 
appear to be especially effective when combined with adequate instructional support (e.g., Atkinson et al., 2000; 
Hilbert & Renkl, 2008; Renkl, 2002; Renkl, & Atkinson, 2007; Stark, 1999; 2000; 2001). Without instructional 
support, the didactic advantages of worked examples cannot fully unfold (Renkl, 1997). 
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1.2 Conceptual Basis of the Integrated Learning Environment 

In order to transfer the instruction-oriented teaching method of learning with worked examples from 
well-structured domains like mathematics to the somewhat ill-structured domain of empirical research methods, 
an adequate contextualizing of worked examples by means of problem-oriented approaches is essential. Stark, 
Kopp, and Fischer (2011) showed beneficial effects of embedding worked examples in a complex integrated 
learning environment on diagnostic competencies of medical students. 

The domain of empirical research methods has a similarly complex structure. Although tests and procedures are 
based on stringent mathematical principles producing precise results, the interpretation of these results is a 
complex process that invariably involves content-related aspects depending upon the particular research 
question. 

For a successful contextualization of instruction-oriented worked examples the use of integrated learning 
environments (e.g., Reinmann & Mandl, 2006) therefore seems promising. 

Integrated learning environments (Reinmann & Mandl, 2006) combine elements from instruction-oriented and 
problem-oriented learning approaches in an effort to maximize their respective advantages and to minimize their 
disadvantages. Learning environments of this type have been shown to support the acquisition of applicable 
knowledge and to have beneficial effects on learning activity and motivation of the learners (Stark, Gruber, 
Renkl, & Mandl 2000; Stark, Herzmann, & Krause, 2010). Against this background, the didactic approach of this 
learning environment puts a strong emphasis on the design principles explained in the next paragraphs. 

Instruction-oriented design principles focus on an instructional orientation and therefore on a systematic 
imparting of knowledge ( instructional design approach; e.g., Reinmann & Mandl, 2006). The degree of 
systematization can be inferred from applying elaboration theory (Reigeluth, 1997). Elaborative sequencing 
aims at systematically depicting learning contents in different degrees of complexity. Elaboration levels of 
increasing complexity resulting from this method allow learners to reconstruct and elaborate learning contents 
step-by-step in multiple levels of differentiation (q.v. Schnotz, 2011). 

Another design principle of this systematization is the arrangement of the worked examples in 
structure-emphasizing example sets (Quilici & Mayer, 1996). An example set of this type is a sequence of 
worked examples characterized by having an analogue underlying structure. With the progression of the 
sequence, complexity increases. This progression intends to support the acquisition of applicable knowledge. 

Problem-oriented design principles require learning processes to be induced by tasks that are both authentic and 
relevant for the learner (Reinmann & Mandl, 2006). This is an important prerequisite for the acquisition of 
applicable knowledge, since learning tasks actually relevant to the learner foster the transfer of knowledge to 
new problems (ibid.). Therefore, authenticity was an essential part of the didactic approach of the learning 
environment (CGTV, 1992; 1993; Grasel, 1997; Grasel & Mandl, 1999; Reusser, 2005). 

Another important problem-oriented design principle considered in the learning environment was the 
presentation of the worked examples in multiple learning contexts and perspectives ( cognitive flexibility 
approach ; Spiro, Feltovich, Jacobson, & Coulson, 1991) to avoid the acquisition of inert knowledge (Stark, 
2000), and to encourage metacognitive processes during learning, such as mindfulness (Salomon & Globerson, 
1987). In this context, Salomon und Globerson (1987) stress the role of mindfulness in successfully acquiring 
and transferring knowledge to new contexts. The concept of mindfulness denotes “[...] a state of mind that is 
defined as the volitional, metacognitively guided employment of non-automatic, usually effort demanding 
processes.” (Salomon & Globerson, 1987, p. 625). There is strong empirical evidence for the relevance of such 
mindful, reflected learning for a deeper processing and understanding of learning contents (e.g., Entwistle & 
Peterson, 2004; Lan, 1998; Wang & Lin, 2007). 

1.3 Conceptional Basis of the Elaboration Intervention 

The elaboration intervention consisted of an advance organizer (Ausubel, 1960) and a subsequent short phase of 
cognitive modelling (CGTV, 1992; 1993) which gave a brief introduction of how to use the advance organizer. 
The advance organizer’s didactic conception is based on the theory of expository teaching (Ausubel, 1960; 1968). 
The vital aspect of expository teaching is to activate prior knowledge by offering associations on the basis of 
pre-structuring and systematization of learning contents. Activating prior knowledge facilitates the assimilation of 
new information into the existent knowledge base or, where appropriate, the adaption of existent knowledge on the 
basis of new information ( accomodation ; Ausubel, 1968; Mayer, 1979; Hasselhorn & Gold, 2006). It also 
facilitates knowledge transfer (Mayer, 1980). According to Ausubel (1968), the advance organizer serves to fill 
gaps between the learner’s prior knowledge and the knowledge needed to initiate the learning process. Several 
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authors have suggested formulating the advance organizer on a concrete rather than abstract level (e.g., Corkill, 
Glover, & Brunning, 1988). 

The didactic conception of the cognitive modelling phase was based on Collins, Brown, and Newman’s (1989) 
cognitive apprenticeship approach; its core was the detailed elaboration of the advance organizer’s basic structure. 
Its intention was to facilitate the acquisition of potentially relevant solution heuristics, and to draw the attention to 
information relevant to the problem. As a result, the cognitive modelling is supposed to facilitate metacognitive 
processes. 

Working with the elaboration intervention should yield a deeper understanding of the learning contents and 
support the acquisition of applicable knowledge, especially regarding complex transfer tasks. Furthermore, the 
elaboration intervention is supposed to enhance mindfulness sensu Salomon and Globerson (1987). 

2. Research Questions 

(1) To what extent is the acquisition of applicable knowledge facilitated by the learning environment and the 
elaboration intervention? 

We expected positive effects of working with the learning environment both with and without elaboration 
intervention on the acquisition of applicable knowledge. Both experimental groups were expected to outperform 
the control group. 

Furthermore, we expected positive effects of instructional support provided by the elaboration intervention on 
the acquisition of applicable knowledge. The experimental group with elaboration intervention was expected to 
outperform the group without elaboration intervention. 

(2) To what extent is the learner’s mindfulness improved by the elaboration intervention? To what extent are 
mindfulness and post-test performance correlated? 

We expected that the elaboration intervention supports learners in working with the learning environment in a 
more reflective, conscious and mindful (Salomon & Globerson, 1987) manner. Therefore, the experimental 
group with elaboration intervention was expected to outperform the other groups. We also expected a positive 
correlation between mindfulness and post-test performance. 

3. Method 

3.1 Sample and Design 

Participants were 70 students of educational science (61 female) recruited in three advanced seminars on 
empirical research methods at a German university. On average, students were in their seventh semester. The 
experimental group with elaboration intervention consisted of 26 (24 female) participants with an average age of 
24.9 (SD = 3.70) years. The experimental group without elaboration intervention consisted of 27 (22 female) 
participants with an average age of 24.2 (SD = 2.89) years. The 17 participants in the control group (15 female) 
had an average age of 24.2 (SD = 2.89) years. 

We compared post-test performance of all three groups within the setting of an experimental field study. The 
participants were randomly assigned to the three groups. The control group received texts on descriptive and 
inferential statistics (e.g., Buhner & Ziegler, 2009). 

3.2 Learning Environment 

The learning environment was a text of 20 pages that included an outline of an experimental study (Tyroller, 
2005) and twelve worked examples referring to the study. 

All worked examples were structured according to the same schema “question - empirical evidence - 
interpretation”: Following the question, a solution was presented. This solution included all the information 
necessary to answer the question. At the end of the text, the solution to each question was summarized in a general 
interpretation section. 

The worked examples were sorted by an ascending level of complexity and arranged in a structure-emphasizing 
example set. For instance, consecutive worked examples were based on the same statistical test (Reigeluth, 1997; 
Quilici & Mayer, 1996). By relating to an actual, real-life study, the worked examples were embedded in a 
context that was both authentic and relevant to the participating educational science students, thereby also 
accommodating motivational goals (CGTV, 1992; 1993). We constructed multiple perspectives on a given 
statistical test by putting the test in different contexts (e.g., mediation- vs. moderation-hypotheses) and by 
presenting different analytical perspectives (e.g., exploratory and confirmatory hypotheses). 
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Before starting work on worked examples, participants were given an introductory explanation about the 
framework, sample, and methods of the Tyroller’s (2005) study. The first four worked examples covered topics of 
descriptive statistics and internal validity. Based on this, the subsequent worked examples were intended to enable 
participants to develop a better understanding of complex statistical concepts. The following three worked 
examples dealt with tests of mean differences. Worked example 8 was a conceptual question and covered the 
operationalization of constructs. The last four worked examples dealt with moderator effects, effect size and the 
resulting practical consequences. SPSS output tables were provided to explain the results of statistical procedures, 
enabling participants to check on internal validity, significance of mean differences, analyses of moderator effects, 
and effect sizes. 

3.3 Elaboration Intervention 

The advance organizer consisted of three text pages and was designed as a structured scheme of twelve guiding 
questions. Each guiding question was directly related to the respective worked example in the learning 
environment and was complemented by a concise description. The additional descriptions explained the learning 
goals of the respective worked examples and emphasized underlying principles. Since the advance organizer was 
formulated with only a minimum level of abstraction, participants were enabled to associate the learning contents 
with known concepts and thus to activate prior knowledge. 

The instructor explained the advance organizer’s use in detail by means of cognitive modelling (CGTV, 1992; 
1993; Collins et al., 1989). Here, students were shown which procedure was useful when working on the guiding 
questions. The instructor made clear how the students could activate their own prior knowledge about statistical 
tests and how they could meet the requirements of the tasks by reading the guiding questions. He interpreted 
exemplified empirical results of the intervention study upon which the learning environment was based. By doing 
so, he explicated his own example elaboration (Stark, 1999). Thus, students were made aware of the metacognitive 
processes involved in finding the solution heuristics (Collins et ah, 1989; Reinmann & Mandl, 2006). Students 
were asked to follow the modelling attentively and to subsequently work on the guiding questions of the advance 
organizer in the same way. 

3.4 Instruments 

3.4.1 Dimensions for Testing Internal Validity 

In order to test the internal validity of the study, the three groups were compared with respect to cognitive, 
metacognitive, motivational, and socio-biographical learning prerequisites that were potentially relevant to the 
learning process. The selection of these criteria was based on several studies in the domain of statistics and 
research methods on learning with worked examples (Krause, 2007; Krause, Stark, & Mandl, 2004; 2009). All 
rating scales had a 6-step rating (from 1 = does not apply at all to 6 = fully applies). All answers to open 
questions were rated by two experienced raters which were blind to the experimental conditions (Cohen’s k 
= .84). Disagreement was solved by discussion with the first author. 

The test on cognitive learning prerequisites comprised prior knowledge in the domain of correlation, /-Test, and 
ANOVA. Students had to answer six open questions. The test proved to be sufficiently reliable (Cronbach’s a 
= .76, theoretical maximum score: 39 points). 

Metacognitive learning prerequisites were tested with a rating scale that consisted of 26 items and was 
sufficiently reliable (e.g.. When I’m learning, I frequently ask myself the question whether or not I have really 
understood what I have just read.”; Cronbach’s a = .90) 

The rating scales for motivational learning prerequisites had already been tested in previous studies on learning 
with worked examples in statistics and research methods (Krause, 2007; Tyroller, 2005). The intrinsic motivation 
regarding statistics and research methods scale consisted of six items (e.g., “I am interested in empirical 
research methods”; Cronbach’s a = .88.). The attitudes towards research methods scale consisted of eleven items 
(e.g., “Statements that have been made by using empirical research methods are absolutely correct”; Cronbach’s 
a = .70.). 

Additionally, participants had to indicate gender, age, and number of semesters. 

3.4.2 Dependent Variables 

We designed a test for applicable knowledge (De Jong & Ferguson-Hessler, 1996) in order to assess the learning 
effectiveness of the learning environment and the elaboration intervention. Applicable knowledge was 
operationalized by using two tasks. 
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Task 1 tested near transfer of applicable knowledge. The task consisted of interpreting the results of an 
experimental study and to deduce implications for practice. The empirical results originated from 1-factorial 
designs. The theoretical maximum score for task 1 was 14 points. 

Task 2 measured far transfer of applicable knowledge; it also consisted of interpreting the results of a study and to 
deduce practical implications. Here, the students’ task was to interpret a 2x2-factorial design so that certain aspects 
of the application of knowledge were much more complex than in task 1. The theoretical maximum score for task 
2 was 15 points. 

Post-test tasks on applicable knowledge showed no significant correlation (r = .18,= .13). 

Additionally, mindfulness was measured using a three-item scale (e.g., While using the learning environment, I 
was completely immersed in my work”; Cronbachs a = .71) which was gathered from studies of Krause (2007) and 
Tyroller (2005). 

3.5 Procedure 

During the first session of the seminar, participants were briefly introduced to the study, and then asked to fill out 
the test on prior knowledge, the scales relating to internal validity, and the socio-biographical questionnaire. 

During the second session of the seminar, the training of the two experimental groups (with and without 
elaboration intervention) was carried out in two separate rooms. Both groups were instructed by experienced 
tutors. 

Each participant had to work on the learning environment by himself. The time limit was 60 minutes for both 
experimental groups, a generous lead time that ensured that all the participants were able to finish working on 
the learning environment. After a short break, participants were asked to fill out the scales on measuring 
mindfulness, and the two post-test tasks. 

Students in the control group worked on the socio-biographical questionnaire, the prior knowledge test, the 
scales relating to internal validity, and the two post-test tasks as well. Instead of working with the learning 
environment, students of the control group had to study selected texts on statistical procedures and their 
interpretation for 60 minutes. Table 1 shows the procedure for all groups and the approximate time provided for 
each test and the training. 


Table 1. Sequence of the tests for both experimental groups 


Test 

Duration (ca.) 

1st session Introductory explanations 

5 min. 

Test of prior knowledge 

45 min. 

Socio-biographical questionnaire; rating scales on metacognition, 
motivation, and attitudes 

35 min. 

2nd session Working on the learning environment/statistical texts 

60 min. 

Test on mindfulness 

10 min. 

Post-test 

45 min. 


3.6 Statistical Analyses 

In order to test internal validity, we conducted y 2 tests and 1-factorial ANOVAs, depending on the data’s 
respective scale level. 

Regarding the two post-test tasks, we formulated directional hypotheses which made it possible to test the effects 
of learning environment and elaboration intervention via 1-factorial ANOVAs with a priori contrasts for three 
groups (e.g., Buhner & Ziegler, 2009). The effect of the elaboration intervention on mindfulness as reported by 
the participants was tested by running a 1-factorial ANOVA. 

As a measure for effect size in our ANOVAs, we used partial eta square (r] p 2 ), and Cohen’s d (Cohen, 1969) for 
contrasts. 
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4. Results 

4.1 Internal Validity 

There were no significant differences for age (F(2,67) = 0.10,/? = .83) nor gender (x 2 (2) = \A\,p = .49) between 
the three groups. The high percentage of female participants was in accordance with gender distribution in other, 
comparable courses of studies in Germany (Statistisches Bundesamt, 2012). The three groups were also 
comparable regarding cognitive prerequisites for learning; we found no significant differences in topic-related 
prior knowledge (F(2,67) = 0.24, p = .79). Likewise, there were no differences in metacognitive and 
motivational learning prerequisites. The scales measuring metacognitive learning prerequisites (F(2,67) = 0.26, p 
= .77) and intrinsic motivation (F(2,61) = 0.54, p = .58) showed no statistically significant differences between 
the groups. Therefore, the current study can be considered internally valid regarding these variables. 

4.2 Effects of Learning Environment and Elaboration Intervention 

(1) To what extent is the acquisition of applicable knowledge facilitated by the learning environment and the 
elaboration intervention? 

Means and standard deviations as well as theoretical maxima are listed in table 2. Performance of task 1 (near 
transfer) was clearly below the theoretical maximum score for all groups. However, we found a significant and 
large effect of the learning environment (F(2,61) = 4.06, p = .022, r\ p 2 =.89, power = .70). As expected, the 
a-priori contrasts showed that both the experimental group with and without elaboration intervention 
outperformed the control group (/(67) = 2.77, p = .007, d = 0.99 and /(67) = 2.22, p = .03, d = 0.76), 
respectively). The contrast effect was large in both cases. In contradiction to our hypothesis, there was no effect 
of the elaboration intervention; both experimental groups performed similarly well (/(67) = -0.64. p = .52). 

Regarding far transfer of applicable knowledge (task 2), all groups performed below the theoretical maximum 
score as well. There was a significant, medium effect of the learning environment ( F(2,61 ) = 9.02, p < •001, r| p 2 
=.78, power = .97). As expected, there was a significant effect of the elaboration intervention: participants of the 
experimental group with elaboration intervention significantly outperformed those without elaboration 
intervention as well as the control group (1(67) = -3.42, p = .001, d = 0.96 and /(67) = 3.85, p < .001, d= 1.15), 
respectively). The contrast effect was large in both cases. The experimental group without elaboration 
intervention and the control group did not differ from each other significantly (1(67) = 0.85,/? = .40). 


Table 2. Post-test performance of the three groups; number of participants, means and standard deviations, 
theoretical maxima 



EG with 

EG without 

Control group 

theoretical 


elaboration 
intervention 
n = 26; M (SD) 

elaboration 
intervention 
n = 21;M(SD) 

n= 17; M (SD) 

maximum 

task 1 (near transfer) 

6.04 (2.58) 

5.61 (2.71) 

3.94 (1.52) 

12 

task 2 (far transfer) 

7.58 (2.80) 

4.96 (2.62) 

4.24 (2.99) 

15 


(2) To what extent is the learner’s mindfulness improved by the elaboration intervention? To what extent are 
mindfulness and post-test performance correlated? 

Mean scores of participants in both experimental groups were clearly above the theoretical mean of the scale. 
The effect of the elaboration intervention was significant and large (F(l,51) = 6.86, p = .012, r| p 2 = .88, power 
= .73). Participants of the experimental group with elaboration intervention (M = 4.69; SD = 0.47) reported 
higher mindfulness than did participants of the experimental group without elaboration intervention (M = 4.19; 
SD = 0.93). 

In contradiction to our hypothesis, mindfulness only correlated significantly with the performance regarding far 
transfer of applicable knowledge (task 1: r = .23,/? = .10; task 2: r = .32,/? = .018.). The correlation was positive 
and rather low. 

5. Discussion 

The results showed that the tested groups were comparable with regard to the socio-biographical, cognitive, 
metacognitive, and motivational learning prerequisites. With regard to these variables that might potentially 
interfere with the learning process, we assume that the current study is internally valid. 
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As the three seminars were mandatory for the students, there were no self-selection-effects influenced by the 
students’ decision concerning seminar choices. The study represents a full inquiry of the respective cohort. 
Therefore, differences in performance between groups as well as differences in mindfulness can actually be 
attributed to the experimental variation. 

Learners accomplished the near transfer task even without the elaboration intervention. On average, participants 
scored half of the achievable points. Thus, regarding the acquisition of the knowledge necessary to solve this 
task, the integrated learning environment has proven to be successful. 

The more challenging far transfer task required a deeper understanding of the presented worked examples. 
Average scores of the participants were about half of the maximum score for this task as well. In accordance with 
the results of Krause et al. (2009), Krause (2007) as well as Krause and Stark (2010), the performance on this 
task was supported by the elaboration intervention. Our results demonstrate once more an issue previously 
discussed in several studies on problem-oriented (Kirschner, Sweller, & Clark, 2006; Schmidt, Loyens, Van Gog, 
& Paas, 2007) and example-based learning (Renkl & Atkinson, 2007; Hilbert & Renkl, 2008; Atkinson et al., 
2000; Stark, 1999; 2000; 2001). The issue concerns the relevance of additional instructional support in complex 
learning. This appears to be especially valid for complex domains with a lower degree of structure. 

Considering especially the high difficulty of the far transfer task as well as the relatively short training phase of 
just 60 minutes, the effects gained by the learning environment and elaboration intervention are considerable. 

Similar to the study of Stark et al. (2009), the elaboration intervention had a positive impact on mindfulness, 
meaning that the participants learned with a higher degree of reflection; the scale indicates a deeper involvement 
with the presented material. The result that only performance on the far transfer task correlated with mindfulness 
indicates that mindfully learning for this complex task proved especially helpful. 

As a whole, our results emphasize the effectiveness of integrated learning environments regarding the acquisition 
of applicable knowledge (e.g., Stark et al., 2011; Stark et al., 2010). Results of this nature are rather scarce in 
several instructional contexts (e.g., Dettermann, 1993). With regard to training in research methods and statistics 
in educational sciences, such effects are in fact an exception (Stark & Mandl, 2000; 2002; Stark, Mandl, Gruber, 
& Renkl, 2002). 

However, there are some limitations of the study. The elaboration intervention was designed as an integrated 
support intervention. The modelling phase referred directly to the advance organizer. Introducing guiding 
questions without the simultaneous use of a modelling phase is pedagogically not well justifiable, especially in a 
field of knowledge that is usually quite problematic for students. For this reason, an isolated test of both 
components by means of experimental variation was not intended. Because of the known learning effectiveness 
of worked examples, the learning environment’s underlying didactic approach was not experimentally varied as 
well. As a consequence, the results can only be attributed to the combination of the described didactic methods. 

Furthermore, the overall sample size was too small to derive more ambitious statements. To state more general 
conclusions, the findings regarding near and far transfer have to be replicated within a study with a bigger 
sample size. The sample should be enlarged regarding participants of other studies of the social sciences 
applying statistical methods like psychology, and sociology. Although the size of the test-power was adequate for 
each test, the study is regarded as a pilot-study to develop an instructional basis for future learning environments. 

The limitations of the current study also suggest desiderata for future research. The learning processes induced 
by the integrated combination of instruction-oriented and problem-oriented approaches need further research in 
order to gain more empirical evidence for optimizing integrated learning environments. In this context, the 
method of thinking aloud (Ericsson & Simon, 1993) that has been well researched in conventional 
example-based learning (Renkl, 1997; Stark, 1999) would appear suitable. In addition, the sustainability of the 
effects gained by the learning approach in general and the elaboration intervention in particular needs to be 
tested in follow-up studies. 

Based on these results, the implemented methodical-didactic approach can be recommended for training in 
research methods and statistics in educational sciences. The principles of the learning environment are also easily 
applicable for other domains with a sufficient consensus on the correctness of a given problem solving task. 
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